DOCUKENT RESUME 



ED 338 696 



TM 017 532 



AUTHOR 
TITLE 

INSTITUTION 

SPONS AGENCY 

REPORT NO 
PUB DATE 
NOTE 

PUB TYPE 



Pace, C. Robert 

Quality r Content r and Context in the Assessment of 

Student Learning and Development in College. 

Center for Research on Evaluation r Standards, and 

Student Testing, Los Angeles, CA. 

Office of Educational Research and Improvement (ED) , 

Washington, DC# 

CSE-R-256 

86 

94p. 

Reports • Evaluative/Feasibility (142) — 
Tests/Evaluation Instruments (160) 



EDRS PRICE MF01/PC04 Plus Postage. 

DESCRIPTORS Academic Achievement; Accountability; ^Content 

Analysis; ^Context Effect; ^Curriculum Evaluation; 
Educational Assessment; ^Educational Quality; 
Evaluation Methods; Graduate Surveys; Higher 
Education; Student Development; ^Student Evaluation; 
« Undergraduate Students 



ABSTRACT 

This report focuses on assessing the quality of 
undergraduate education. A framework for thinking about the topic is 
provided, followed by a discussion of the ways in which the 
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typically measured, with suggestions for new indicators and measures. 
The significance of contexts and processes is noted, with 10 
suggested measures of process and context variables* Suggestions are 
also made for improving the content of alumni surveys. How these 
quality-related issues are viewed in relation to concerns for 
accountability, standards, evaluation, amd assessment is briefly 
discussed, with renewed emphasis on how assessment can best 
contribute to learning and improvement. An adequate assessment of 
educational and instructional quality requires a battery of measures, 
a variety of observations, and the recognition of multiple alms# In 
addition, the role of students in evaluations needs to be given 
greater attention; by keeping time logs, diaries, and writing an 
autobiography, students can directly express and document both the 
process and results of learning. Two tcibles and 20 figures are 
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ABSTRACT 



This report focuses on assessing the quality of undergraduate 
education. A framework for thinking aoout the topic is provided, 
followed by a discussion of the ways in which educational context 
is typically organized and student achievement is typically mea- 
sured, with suggestions for new indicators and measures. Of edu- 
cational and instructional quality, a systematic and comprehensive 
inventory of student activities is currently. available. Then the 
significance of contexts and processes is noted, with ten suggested 
measures of process and context variables. Moving beyond the college 
years, suggestions are made for enriching the content of alumni 
surveys. The paper concludes with a discussion of quality issues 
in re1ati''n to concerns for accountability, standards, evaluation 
and assessment, with emphasis on how a^ssessment can best contribute 
to learning and i improvement. 
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INTRODUCTION 



Quality and excellence are such vddely used words In a11 the 
recent reports about the condition of higher education and the 
Importance of assessment that there Is little merit In reaffirming 
the obvious virtues intended by those words. Everyone wants good 
quality at good colleges and good evidence to prove the goodness. 

There Is, nevertheless, some merit in sorting out. what various 
advocates of quality mean by quality. My own view Is that It is 
useful to distinguish between Institutional quality, educational 
quality, and Instructional quality. It Is also useful to be clear 
about "quality for whom?" Is It for undergraduate lear^'lng and 
development? Is It for graduate and professional school? Is It 
for adult education? Is It for contribution to research and 
Inventions? No one Indicator of quality Is sufficient or relevant 
for all these aspects of higher education. Moreover, It Is also 
Important to view quality In relation to purposes. How well Is one 
achieving what one Intends to achieve? There Is no single goal or 
purpose that Is equally Important or equally emphasized by all 
colleges and universities. 

It Is true that all colleges and universities teach 
undergraduates, but It does not follow that the quality of 
undergraduate learning Is an equally appropriate criterion for 
judging Institutional quality at all types of Institutions. 
It Is also true all colleges seek and have faculty members who are 
judged to be scholars. But it does not follow fran this that all 
colleges should be rated by the scholarly and research eminence of 
their faculty members. At some colleges faculty members are 
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expected to make significant published contributions to research, 
and their expenditure of time Is arranged to enhance that outcome; 
but at other colleges faculty members are expected to devote a 
large share of their time to stimulating the Intellectual and 
personality deve1(^ment of every undergraduate student. 

The facts of Institutional diversity In American higher 
education require diversity In the Indicators for judging quality. 
For example, responses of faculty members rating the Importance of 
tvfenty goal areas (fran the Institutional Goals Inventory, 
published by ETS) at various Institutions In California showed 
research and advanced training as the two most Important goals at 
the University of California but as the two least Important goals 
at private four-year colleges. Also, the goal area described as 
Individual personal development ranked third from the top at the 
private four-year colleges, but fourteenth at the University of 
California campuses. 

The large differences In alms and activities at big 
research-oriented universities compared to small liberal arts 
colleges suggest the scope of relevant Indicators needed to judge 
Institutional quality. With respect to the assessment of 
Instruction and the educational experience of undergraduates, 
however, the relevant Indicators of quality are similar at large 
universities and at small colleges. The differences are In the 
context, the environment, the conditions, the priorities that may 
be related to the quality of undergraduate student learning and 
devel opment. 



The following lists illustrate how the scope of inquiry and 
indicators of interest change as one moves fran the assesment of 
Instructional quality, to educational quality, and to institutional 
quality. 

Instructional quality for undergraduates can be judged by: 

1. The content of courses, curricula, and requirements 

2. The expectations of faculty for student performance 

3. Supportive contacts between students and faculty 

4. The time spent on school work 

5. The requirements for reading, writing, and library use 

6. The flexibility of teaching-learning methods in relation 
to the characteristics of students 

7. Evidence of student progress and level of attainment in 
knowledge, skills, and other attributes related to the 
objectives of courses, curricula, and programs 
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Educational quality for undergraduates can be judged by: 

1. The facilities and opportunities provided by the college 
for student learning and development beyond the classroom 
- clubs and organizations, the student union, athletic and 
recreational facilities, residence facilities, counseling 
and health services, etc. 

2. The amount and quality of effort students Invest in 
capitalizing on the above facilities and opportunities 

3. The general level and vitality of the student body - 
Intellectually and socially 

4. The supportiveness of students' relationships vrlth one 
another 

5. Evidence of student gains with respect to such outcanes as 
clarifying values and ethical standards, 

self -understanding, understanding others, teamwork, 
awareness of and concern about social problems that 
transcend the boundaries of specific courses 

6. General satisfaction with the college experience - 
intellectually, socially, and vocationally 

7. And, all the previously listed indicators of 
instructional quality 
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Institutional quality In doctoral granting universities can be 
judged by: 

1. Research and scholarly eminence of the faculty 

2. Financial resources for research 

3. Contributions to research 

4. Services to business. Industry, agriculture, and the 
professions 

5. Other services as exemplified by a teaching hospital, 
dental clinic, psychological clinic, etc. 

5. Contribution to the arts, as In musical concerts, plays, 
art exhibits, etc. 

6. Quality of the library, laboratories, etc. for research In 
special areas 

7. Quality of the athletic program 

8. Adult education as In extension courses and In the 
excellence of publications by the University Press 

9. Special centers of scholarship — as Illustrated by the 
Hoover Institute at Stanford, the Kennedly Center at 
Harvard, the Humphrey Center at Minnesota, the Scrlpps 
Institute at UC San DIegOt etc 

10. Training for the professions and advanced scholarship In 
the academic disciplines 

11. Quality of the educational resources and opportunities for 
undergraduates 

12. Quality of undergraduate teaching and course content 

13. And, all the previously listed indicators of Instructional 
and educational quality for undergraduates 
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The foregoing lists are Intended as Illustrative rather than 
as a definitive classification. The Important point is that there 
are many helpful and valid ways to describe Institutional, 
educational, and Instructional quality. My own view is that these 
are three distinguishable levels, and that, like a scale, the 
smaller levels are also parts of the larger ones. 



Institutional Quality 
(all aspects of an Institution's activities and alms) 



Educational Quality 
(The content, context, and Investment In 
the total "experience of education") 



Instructional Quality 
(The content, process, and product 
of courses, curricula, and teaching) 



Instructional quality Is part of a larger dcmaln called educational 
quality, and both. In turn are part of the still larger domain of 
Institutional quality. 
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The focus of the present report is not on the large domain of 
institutional quality, its focus is on educational and instructional 
quality, and more specifically on assessing the quality of the educa- 
tion of undergraduates. The report begins with a framework for 
thinking about the topic. This is followed by a discussion of the 
ways in which educational content is typically organized and student 
achievement is typically measured; plus suggestions for new measures. 
Then the significance of contexts and processes is noted, with ten 
suggested measures of process and context variables. Finally, moving 
beyond the college years, suggestions are made for enriching the content 
of alumni surveys. How all these quality issues are viewed in relation 
to concerns for accountability, standards, evaluation and assessment 
is briefly discussed in a final section, with renewed emphasis on how 
assessment can best contribute to learning and improvement. 
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A FRAMEWORK for EVALUATING the QUALITY of 
STUDENT LEARNING and DEVELOPMENT In COLLEGE 



A study of student development and college Influence requires two 
parallel lines of Inquiry — one focused on the students (the Individual) 
and one focused on the college (the Institution). In the overview outline 
below, I give examples of these lines of Inquiry and Indicate relevant 
sources of student and Institutional data. Following the outline, with Its 
sequence of leading questions, each topic Is Illustrated more fully. 

Overview Outline 



The Question 
Who goes? 



The Individual 

Characteristics of 
entering students 



The Institution 

Admissions policy 
and practices 



What do they do 
after they get 
there? 



Student Involvement In 
using the facilities 
and opportunities for 
education 



Availability of 

resources 
Courses and curricula 

requirements 



What's It like? 



Student perceptions of 
the environmental 
press 

Academic, social, 
administrative 



Faculty expectations 
Physical arrangements 

Processes of 
Instruction 
Support services 



What do they get 
out of It? 



Evidence of students' 
knowledge and 
understanding. 
Intellectual skills, 
occupational skills, 
attitudes. Interests, 
values 



Graduation, grades, 
courses completed 



And after college, 
what? 



Alumni status 
Satisfaction 
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Who goes? 

Characteristics of entering students can be surveyed by using 
act's Entering Student Survey. This Includes background Information, 
educational plans and preferences, college Impressions, and a place 
for 30 additional questions. This additional space could be used for 
any number of purposes: 1) one could ask questions about values, 
etc., that would overlap with selected Items on the Student 
Information Form (annual freshman survey conducted by the Higher 
Education Research Institute at UCLA); 2) one could locally develop a 
set of Items about other matters of special relevance to the college. 
Having the local faculty and staff develop items of local relevance 
would be a worthwhile activity and would generate Interest In the 
results. In addition to questions about beliefs and values one may 
want to know more about the students' previous activities, family 
background, aspirations, etc. 

If the college uses the ACT rather than the SAT as an admissions 
test. It then also has a measure of the student's entering achievement 
level In the basic fields of English, math, sciences, and social 
science and history. 

Basically, the college needs to know where the students ccme 
fran, why they cane, what they expect; and especially the Initial 
status of their knowledge, beliefs, values, etc., that the college 
hopes to Influence so that one has an Initial reference point for 
subsequently judging their growth and development* It Is this latter 
element of the question. Who goes?, that Is most frequently 
unanswered. There are, nevertheless, some measures of students' 
Initial status that could be used more systematically In an evaluation 
or assessment program. Many colleges require an English compusltlon 
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at entrance and could require one at exit. The subject of the 
composition could be one that would reveal students' knowledge about 
the subject, or ability to analyze an argument, or similar Insight. 
The essay could be graded as a "composition", but then also graded by 
other judges for the level of understanding expressed. A senior esse\y 
on the same or comparable topic could also be graded In this dual 
manner. Also, some students will have taken College Board achievement 
tests which serve as measures of entering skills and knowledge. 
Additionally, there are, of course, various measures of attitudes, 
interest, values, and personality traits that could be given to 
samples of students. If not to all students. In using such measures 
as a baseline for subsequently observing change, one needs to be clear 
that what Is measured Is changeable (rather than a stable trait) and 
that change In a particular direction is Intended by the goals and the 
program of the college. 

On the Institutional side, the answers to the question "Who 
goes?" win be Influenced by admissions policy, recruitment 
activities, pamphlets, etc. Also by what the college offers and what 
it says it stands for. What are the features that attract students? 
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What do they do after they get there? 

Pace's College Student Experiences Questionnaire (CSEQ) Is the 
most systematic and canprehenslve Inventory of student activities 
(Involvement) currently available. It measures the quality of effort 
students put Into using the facilities and opportunities In the 
college that are Intended for their learning and development — 
academic facilities such as the library, laboratory, courses, cultural 
facilities, contacts with faculty, writing experiences, clubs and 
organizations, breadth and Cepth of student acquaintances and 
conversations, etc. (see Pace, CSEQ; TEST MANUAL & NORMS. UCLA Center 
for the Study of Evaluation, 1987). 

One section of the ACT Student Opinion Survey has a list of 23 
college services or programs and asks students If they have used the 
service and whether they wei^ satisfied with It. It also has space 
for 30 locally developed questions. 

Pace's CSEQ reveals the frequency and scope of student activity 
as well as its quality; but neither that Instrument nor the ACT Survey 
deals with what one might call the clinical experience of college. 
The experience of college also Involves personal /social frustrations 
and satisfactions, depressions and elatlons, various encounters that 
may be personality enhancing or regressive. We need a theory-based 
measure for these aspects of what happens between entrance and exit. 

Courses and curricula are major determinants of what students do 
in college but we need more sophisticated ways to describe and 
classify them. Transcripts provide course titles but are much less 
useful In defining course content. Catalogues describe course 
requirements and curricula but are not accurate Indicators of what 
students actually take. Course content Is also connected to teaching 
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methods, but transcripts and catalogues do not reveal this 
connection. A sophisticated answer to the question "What do they do 
after they get there?" would describe how the student Interacts with 
the courses and curricula. What Is the "experience" of the course for 
the student? 

The Institutional side of the question "What do they do after 
they get there?" would note what facilities and resources the 
institution actually provides. It would also note any requirements 
regarding the use of facilities, Including courses, curricula, 
credits, etc. 



What's It like? 

This question refers to characteristics of the college 
environment and how students perceive It and respond to It. In the 
CSEQ there are eight rating scales about different aspects of the 
environment Its perceived emphasis on academic, esthetic, critical, 
vocational, and personally relevant matters, anri the perceived 
supportlveness and congenial relationships among students, between 
students and faculty, and with the administration. 

The ACT Student Opinion Survey has a section about the college 
environment which consists of 42 statements grouped under topics 
labeled academic, admissions, rules and regulations, facilities, 
registration, and general — statements to which students respond 
by Indicating their level of satisfaction. There Is also, as In other 
ACT Surveys, an opportunity for the college to ask up to 30 additional 
questi ons. 
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ACT also has a Withdrawing/Nonreturning Student Survey. This 
Includes Items about reasons for leaving — personal, academic, 
emotional, financial, and employment. It also has a section about 
students' satisfaction with college services and with various college 
characteristics. There is also space for up to 30 additional 
questions. 

In general, student retention/withdrawal can be viewed as the 
degree of "fit" between the person and the environment in several 
basic respects the level and scope of the academic offerings, the 
compatibility of the student and the campus social life, and the 
vocational expectations of the students. 

The institutional side cf the question "What's it like?" consists 
of what the institution does to create the campus climate. This 
would include faculty expectations for students' academic performance, 
expectations and perhaps rules about proper student conduct, course 
requirements, various formal and informal processes of instruction, 
the physical arrangement of the campus, architecture, landscaping, and 
other elements that help to identify the character of the college and 
to define "what's it like" to be a student there. 

What do they get out of it? 

College records will show that the student got a bachelor 
degree, completed a specific list of courses and the grades received 
in those courses. 
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The CSEQ has a list of 21 Important goals of higher education, 
with the students indicating how much gain/progress they feel they 
have made as a result of their college experiences. These outcomes 
include various intellectual skills, breadth and depth of subject- 
matter learning, vocational preparation, and personal /social 
development. The student reports correlate highly with other 
indicators of achievement. 

Ideally one should have a battery of achievement tests, and 
measures of interests, beliefs, and values, etc* relevant to the 
college program and purpose. These tests, or ? orellminary version of 
them, would have been given to entering students so that differences 
In responses between entrance and exit could then reflect the 
influence of the college experience on such knowledge, values, etc. 

We have some instrumentation for such comparisons with respect to 
knowledge and understanding of the major academic disciplines — 
sciences, social sciences and history, humanities and arts, etc. 
Tests exist but many are not currently available for this purpose and 
they are much too expensive. For students aspiring to enter graduate 
school in a department that requires one of the 6RE special field 
tests, or to gain certification or licensing in certain fields, one 
could presumably see how well the students do compared with student'^ 
elsewhere. 

There are published measures of values that could be used at 
entrance and exit — the All port- Vernon-Lindzey Study of Values, 
Rokeach's Instrumental and Terminal Values scales, the Omnibus 
Personality Inventory, etc. 

Also, if the CSEQ is given to a cross-section of students at all 
levels, one can compare the progress reported by students at the end 
of the freshman year with progress reported at the end of the senior 
year. ^ 2 0 



After college, what? 

ACT has an alumni survey questionnaire, Intended for recent 
graduates. It has questions about continuing education, retrospective 
opinions about college benefits and satisfaction, and employment 
history. Presumably the alumni office of the college will have 
continuing contacts with the graduates. 

We need an alumni questionnaire Intended for graduates after 20 
years or so seeking Insights regarding the quality of their lives 
and their Involvement In civic, cultural, intellectual, and similar 
aspects of adult life that are relevant to the purposes of higher 
education. 
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Process and Context as Outcomes 



Some answers to all the above questions Who goes? What do 
they do after they get there? What's It like? What do they get out 
of It? — can be obtained from student responses to the CSEQ. The 
CSEQ not only measures the quality of outcomes (students progress 

* 

towar^d Important educational objectives), but also the quality of 
educational processes (frequency, scope, and level of student 
activity /effort In using the facilities and opportunities for learning 
and development in college), and the quality of the educational 
context at the college (characteristics of the educational 
environment). Usually one regards student achievement/gain as the 
appropriate dependent or outcome variable, and then seeks to find out 
hoM various background factors, process factors, and context factors 
contribute to the desired outcomes. It Is worth noting, however, that 
one can also regard process and context elements as the dependent or 
criterion measures. For example, a college might say that It seeks to 
create and facilitate an undergraduate campus experience characterized 
by high levels of student Involvement and effort. Scores on the CSEQ 
activity scales would reflect how well this goal was being attained. 
Similarly, a college might saiy that It wishes to have an environment 
that Is scholarly, esthetic, and supportive. Student ratings of the 
environment would indicate whether the college has been sucessful In 
creating such an atmosphere. 
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other Models 



The sequence of questions presented here Is one model. It is 
similar In many ways to a model I presented In a previous publication 
(Pace, Measuring Outcomes of College , Jossey-Bass, 1979) as follows: 

• Entrance 

criterion measures at entrance, student characteristics 

^ College Experiences and Events 
facilities and opportunities 

• Student Quality of Effort 

In using the facilities and opportunities 

College Environment 
environmental press 

• Exit 

Student development and college Impress as Indicated by 
differences between criterion scores at entrance and 
exit, self -ratings of progress, benefits, satisfactions, 
attitudes toward college, and subsequently, evidence fran 
alumni studies of continued Interests, continued 
learning, etc. 

A general causal model for assessing the effects of differential 
college environments on student learning and cognitive development was 
presented by Pascarella In a 1985 publication. (Pascarella, "College 
environmental Influences on learning and cognitive development", pages 
1-61 In Higher Education; Handbook of Theory and Research (J. Smart, 
Editor), Agathon Press, 1985). The model begins with Student 
Background/Pre-College Traits cind with Structural /Organl£':it1onal 
Characteristics of Institutions. Then there are three subsequent 
Interrelated elements labeled Interactions with Agents of 
Socialization (e.g. faculty, peers). Institutional Environment, and 
Quality of Student Effort. All of the above then have some effect on 
Learning and Cognitive Development. 



It Is Interesting to note that the Pascarella model deals with 
all the sequential questions I have suggested as a framework for 
evaluation. Student background and pre-college traits answer the 
question. Who goes? Structural /orga»i1zat1onal characteristics of 
institutions and the institutional environment are concerned with the 
question, What's it like? Interactions with agents of socialization 
and the quality of student effort are answers to the question. What do 
they do after they get there? And, learning and cognitive development 
deals with one major element cf the question. What do they get out of 
it? 

In the earlier Pace model and in Pascarella's model there is one 
highly Important variable that is not explicitly identified and 
emphasized, although it is doubtless Implied. This is the content of 
courses and the curriculum. The next part of this report focuses on 
educational content and measures of student achievement related to 
that content. 
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EXPANDING THE RANGE OF CONTENT MEASURES 



HoM educational content Is defined and put together In courses 
and curricula determines, to a very great extent, how students' 
achievement should be measured. The question Is quite simple: How 
well are students learning what Is taught? But the answers are not 
simple because the selection and organization of content Is complex 
and serves a variety of purposes which may have relatively little 
overlap. Some existing tests fit seme educational content quite 
adequately. For other ways of organizing content and defining 
curricula, there my not be a good fit with existing outccme 
measures. Moreover, the simple question of how well students are 
learning what Is taught Is not sufficiently encompassing. We may also 
be Interested In students' possession of knowledge, concepts, and 
skills that we regard as Important In higher education Irrespective of 
what courses the students (lave taken* 

Three questions are relevant to discussing measures of 
educational content. First, how Is the content organized? In most 
college courses content Is organized by academic disciplines. It may 
also, however, be organized by the goal of developing competence or 
skill In some activity. Second, what Is the level of content and 
measurement? It may be a specific course taught by a specific 
teacher. It mdy be a fairly well defined set of content (an economics 
course, for example) but taught by different teachers to different 
students at different colleges. Or, ' may be a cluster of content 
rather than a specific subject, as Illustrated by general tests In the 
sciences, the social studies, humanities, etc. Third, what Is the 
level of generalization one wants to make from the use of various 
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measures? It might be a generalization about a particular student or 
group of students. It may be generalizations about a course, or 
group of courses defined as a program or a curriculum. It ma> be the 
college, as In the level and scope of student achievement compared 
with other colleges. Or, finally. It may be generalizations about the 
quality of higher education In the United States. 

Most courses and curricula In most colleges are organized by 
academic disciplines. An academic discipline Is a bo6y of knowledge, 
concepts, and methods of Inquiry characteristic of a particular field 
of knowledge. The knowledge, concepts, and methods of chemistry are 
different fran the knowledge, concepts and methods of psychology. One 
might also distinguish between a discipline and a subject-matter. 
Econanlcs Is a discipline; money and banking Is a subject-matter. One 
can apply this distinction to the administrative organization seen In 
most colleges. The departnent Is named for the discipline: the 
economics department, for example. Then various groupings of 
subject-matter are the names of courses: International trade, for 
example. 

Daniel Bell holds that there are basic differences In the nature 
of Inquiry between broad fields of knowledge, as Illustrated In the 
following quote. 

different principles govern the acquisition of knowledge 
In the sciences, the social sciences, and the humanities ... 
In the sciences (and In mathematics), the learning is 
sequential : within any science stipulated levels of 
prerequisites define the kinds of knowledge necessary before 
one can proceed to the next level. In the social sciences, 
the pattern Is one of linkages between fields. Elements of 



economic policy, for example, are understandable only In a 
political context, and this In turn Is dependent upon some 
conception of the social community. In the humanities, 
knowledge Is concentric ; one moves within many different 
circles of meaning In the effort to attain. If ever, an 
understanding of a text and an experience." (p. 174 In 
Daniel Bell, The Reforming of General Education . Columbia 
University Press, 1966). 

At UCLA, the nature of content In various fields of study 
(academic departments offering an undergraduate major) Is described In 
"conceptual diagrams" for the benefit of new students and made available 
as one part of an Advising Workbook. I have selected four such diagrams 
for Inclusion In the present report -- one from science, one from social 
science, one from humanities, and one from a social problems topic. 
These diagrams were prepared by the faculty. Their significance for 
understanding content Is well described in the Introduction to this 
section of the Advising Workbook, as follows: 

"The diagrams ... are Intended to visually represent the 
way In which Inquiry — the questions of the disciplines 
and majors — Is organized within departments. The 
questlon(s) at the top of the diagrams are those which 
define the academic field In the broadest possible waty. 
These central questions are both extremely simple and 
extremely profound. In the phrasing of these questions, 
scholars have carved out a chunk of human Inquiry and 
called It their Intellectual territory or academic 
discipline." 

a? 
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"The next level of questions In the diagrams emanates from 
both the Information that has been established to date and 
the directions In vyhlch discovery and thought about the 
subject are aimed. These are the subflelds of the majors, 
and they serve to organize further inquiry into manageable 
pieces. The third level questions flow from the second 
level. In many cases, these are the questions that are 
actually being asked by UCLA faculty through their ongoing 
research. Collectively, these questions represent the 
frontiers, the borders and the boundaries of what we now 
know and are seeking to understand better." 
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PHYSICS 



What are the laws of nature that govern the behavior of matter from the most 
microscopic particles to the large scale structure of the universe? How do 
you expmss these laws mathematically? How do you discover and verify them 
experimentally? How are the laws of nature related to one another? F6r ex- 
ample, how Is electricity related to magnetism and light? 



I ELEMENTARY PARTICLES, 

\^ NUCLEAR AND ATOMIC PHYSICS 

' What are the basic mathematical laws that 
govern the number and prypertles of the 
elementary particles? 



UQUID8, BIASES AND PLASMAS 

What is the Interrelation between atoms that 
make a substance a liquid rather than a gas 
or 8 solid? 



What ptrtlclei maxs up the 
itomle nueieuil What ttt 
(host ptrtielta mad* of? 



How do atoms Inttrset with 
f ach othar and wtth tlghtt 



How do Rqirida flow and wh^ 
can thaat ttqulda bt tu^ 
bultnt? 



'^0 



SOLID STATE PHYSICS 

In what patterns do atoms comblnt to form 
crystalSi polymers, and more exotic 
substances? How do these patterns depend 
on temperature? 



Csn wt usf ststfstlctf pro» 
pertles of $ tystsrr cf many 
(ndtptncfcnt atoms to 
dstcfilss gt stsT 



Nov do thtsa subsuneti 
conduct olcctrfcHy and 
htat? What are ssmi- 
conductors and super- 
conductors? 



Why ana soma magnatfc and 
othtrt not? 
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What happf ni whan easts 
iMcoma ao hot that thair 
atom* t>acome Ionized and 
they conduct clf ctrlcUy? 



BEST COPY MiiiBLE 



PHILOSOPHY 

How can one apply rigorous conceptual analysis to the many Issues of human 
thought and life which resist solution by the methods of other academic 
disciplines? 



HISTORY OF 
PHILOSOPHY 

)W have the ereat philosophers of the 
tt stated and tried to answer the main 
oblems of philosophy? 



It Inslghtt Into tht 
ir* of virtut can wt 
I from th» (tudy of 
0 tnd Aristotlt? 



LOGIC, SEMANTICS 
ANO THE PHILOSOPHY 
OF SCIENCE 

What Is the nature of logical reasoning 
(both formal and Informal), the meaning 
communicated by words and linguistic 
structure, and the assumptions, 
worldvlewa and methodology of science? 



ETHICS AND 
VALUE THEORY 

On what grounds can we determine 
whether an action, attitude, or social 
system Is morally good or bad? What Is the 
a esthetic value of an artistic work or per* 
f&rmance? 



METAPHYSICS AND 
EPISTSMOLOQY 

What are the fundamental leatures of reaH 
ty? How can beliefs be lustlfied? Can w 
know anything at all? 



Did Augatina and Anaalm 
•uccead In showInQ that 
ihe axiattnca of God 
lollowt from the vary 
natura of truth? 



Ara all logical truths 
provabltf 



What II the natura of 
aclantitle axplanatlon? 



What la fraadomt 



Doaa iha and luatlfy tha 
maana? Is It alway* right 
to do tht thing that wlU 
havt tht btst const* 
quancts? 



What It Iha raiatlonahip 
balwaan mind and twdy? 
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Old Kant succttd In 
titmOnitntlnB thai, %yf%n 
In tht ciSi ot our own 
mlRds« if^t cannot havt 
any knowladga of thinfit 
at thay ara tnthamaatvaa, 
ul only ol appaarancaat 



Ara unobaarvabia 
phyalcal partlclaa raal 
ilUnga or uaatui tiellons? 



Why ahould ona tatl tha 
tfulHt 



What la tha ralationthlp 
batwaan ratlglon and 
athlca> 



In whit atnsa am I tt 
tama parson i waa In 
yaara agot 



Ara wa {ustlfltd In btm^ 
cartaln about our own 
mantal itatai? 



Ad 



ECONOMICS 



r 



THEORY, METHODS, 
MATHEMATICS AND 
STATISTICS 

What theoretical modeli of the 
eeonomlo ayttem and proeeaa 
can help ua to underattnd and 
predict economic goala and out- 
cornea, and how can 
mathematlca and atatlstlea be 
uaed to model, meaaure, and 
predict the changes in a perfor> 
manee of an economic ayateni? 



Hjw egn tvotvlna modiU In 
tfionomlet •nd taw AND eon> 
tllel vtrtu* eooptrillon 
•trttt^let b« compered? 



How do people choose to spend their Incomes? How do they decide on the 
best use of their labor end other resources? Whet determines the comparative 
wealth of nations end rates of growth and decline? 



URSAN AND 
REQIONAU 
ECONOMICS 

What laauea are ralaed In 

eeonomlo theory and analyale 
when applied to urban areaa 
where iasues of tranaportatlon, 
houaing, and tabor marketa are 
o1 prime Imporjance? 

How Co labor laws afttet the 
wtllaro ol munletpal 
workara? 



PUBLIC 
FINANCE 

What are the roles and reapon> 
albllltlea of national, atate. and 
local governments In ttrma of 
taxation, public spending, and 
fiscal management? 



What ara the eharaeltrlstlcfl 
ol a monatary authority that 
will laad to a aoclally optimal 
Inllatlon rata? 



ECONOMIC 
INSTITUTIONS 

What are the historical origlna 
and history of the present 
economic system and Inslitu* 
tlons such as banking, private 
business, labor unions, and in- 
dustry? 



What a?e aspactt ol lha aariy 
(nduairlalUatlon in tha 
Amarican Nonhwasi? 



LABOR 
AND 
ECONOMICS 

What Is the nature of the relatio 
ahip bttween labor, wage 
business cycles, unemptoymer 
tabor unions, private Industr 
and public (governmsntr 
policy? 



What la tha rclallenahip bal< 
waan Invanterlea ol goods 
arfd tht layotl ol workers over 
the business cycle? 




INiSUSTRIAL 
ORGANIZATION 

i^ow are eompanlea, corpora- 
tions, partnerships, and other 
business firms structured and 
organized end what la their role 
In an economy' 



What Is the nsture and p.'o- 
cess ol tha separation ol 
ewnerthtp and control in the 
modern corporation? 



MONEY 
AND 
BANKING 

Where does money come from 
and how is Its supply regulated? 
How does a banking system 
work and what la a bank's 
relationship wHh inflation, 
intereat. and an economy? 



What deteffnines the Intereat 
rait paid on International 
bans thst sre backed by tht 
oovemmenl ol the borrower 
and there appear* to be a pro- 
babutiy ot deiavU? 



INTERNATIONAL 
ECONOMICS 

How are the economics of ns- 
ttons linked through Intema- 
tlonal trade, commerce, and 
monatary policy? 



Is a country's trcdlnfl rels< 
tlons with othe/ eounlrtes s 
tuncUon ef iu own produc 
live Isctoro? 



EdSONOMIC 
DEVELOPMENT 

How and why do nittonal 
economicie grow or not grow and 
how can daveloping countrloa 
bast achitve thair economic 
goala? 



I 



How do differing ssti of 
irtonetsry sieliftnst nilti il» 
kiCX th* p«ftomi«net of tht 
tconomSftS of va/ious UUn 
AmtflO«fi countrUt? 
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WOMEN'S STUDIES 



What Is and what hat been the female experience? How does the cultural 
sex/gender system affect our institutions, our culture, our interpersonal rela- 
tionships, our personal development, <our employment and family structures, 
and our systems of knowledge? How does studying the world from a female 
perspective affect the questions, methods, conduct and conclusions of 
research? 



LITERATURE 

How have women been por- 
trayed In literature? What are the 
themes, forms, and techniques 
of women's literaturot how are 
these expressed In characters, 
symbols, and perspectives, and 
how have they evolved? 



HISTORY 

What has been the nature of 
women's lives and contributions 
in traditional and modern 
societies? How and why have the 
social, economic and political 
roles and conditions of 
womanhood evolved? 



3^ 



r 



PSYCHOLOGY 

How does the existence of 
biological sex and a cultural 
sex/gender system affect 
women's end men's experl* 
enees, behaviors, perceptions, 
evaluations, and Intcroctlons? 
How may understanding sex and 
gender contribute to our 
understanding of the mental end 
emotional forces and processes 
of female and male experience? 
How may studying women's ex- 
perience Inform our knowledge 
of human development and per- 
sonality? 



HUMANITIES AND 
SOCIAL SCIENCSS 

How do specific theories and 
Ideas about human experience 
and Its expression change when 
women are the focus and center 
of the inquiry? 



itt !• the nctur* and 
ig» of English and 
itrlein lllerary works 
and about women? 



r 



J L 



What If tht naiurt and 
rangt of French ItUratura 
by and about womtn? 



What «r« tha socio* 
hiatOffcal foundation) of 
Chlntsa, Eurnpaan, flua* 
alan and Wast African 
women'a political and 
cconomie actions? 



What art the demo* 
Sraphfe, tconomfc« lociaf 
and Inteitactua) condl* 
tlorts that havf shaped 
womtn*3 lives Jn diflarenl 
periods of Amarlcan 
history? 



What art the cffeeti of 
job discrimination, token* 
Ism, chanaing work roles 
and duaNcareer mar* 
rlayea on female pay* 
ohology and the quality of 
women's Uvea? 



How do sex and gender 
affect personality dif* 
ferences. Inteii^ctuat 
achievement and aoclcl 
inleractlon? 



What la the relationship 
bei.vetn traditional 
ethnographic theory and 
the female roles which 
have been emphasized In 
the social and an* 
(hropologlcal literature on 
traditlQniJ and contem* 
porary societies? 
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What are the eoolal 
psychological, polltica 
and economic foretf 
which impact upon the In 
terpersonal relationshlpf 
of eiaek women aj 
members of thaii 
biological and ethnic 
groupa? 



38 



[ST COPY BkUm 



How Is women's status 
and role expressed In 
Oreelci ftoman and Early 
Christian myth, an and 
Utf ralure? 



One can see in these conceptual diagrams that the content of 
physics and of philosophy is mainly structural. Economics, on the 
other hand, reveals a division with nine sub-topics, with some link- 
ages between them. One can also see, from the various questions, 
that economics has many linkages with other social studies — especially 
political science, sociology, and psychology. Women's studies has a 
set of questions that are similar to many social problems or integrative 
fields — namely viewing the topic from several different perspectives: 
from literature, history, psychology, and humanities and social sciences. 

The last of these conceptual diagrams, women's studies, represents 
a different way of organizing content. There may be a growing interest 
in interdisciplinary courses, and in programs related tc basic social 
issues. For example, one of the most popular undergraduate majors at 
Stanford is "human biology." At the College of the Atlantic in Maine 
the entire curriculum is organized around the broad topic of "ecology." 
These special combinations are a third, and perhaps increasingly pop- 
ular, way of organizing content; but we do not presently have 
standardized tests for that content. 

In addition to the organization by academic disciplines, and by 
social issues or problems, many of the courses in professional schools 
represent a third and very common basis for determining content — 
namely, the development of some skill or competence. This is seen in 
such fields as education, engineering, medicine, dentistry, aspects of 
architecture, social work, nursing, business, performing arts, athletic 
activities, and a host of skilled trades. • 
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For the most comnon ways of organizing content there are many 
published achievement tests. For example, both Educational Testing 
Service and the American College Testing Program prepare tests for 
certification or 1 icensing in various fields — a certified public 
accountant, a license to fly an airplane, or sell real estate, or 
operate a beauty parlor, or teach school. In medicine, dentistry, 
nursing, teaching, or performing art, etc. the ultimate test Is the 
direct observation of performance or skill. Paper and pencil tests can 
describe a situation or case and ask "what would you do In this case?" 
But that Is not an adequate substitute for observing the actual 
behavior. For content organized by academic disciplines there are many 
achievement tests. This Is the most coimon type of test. Unfortunately 
many of those good tests are available only for seme special purpose and 
are not, at the present time, available for the general purposes of 
evaluation and assessment of programs. Some colleges require students 
to take one or more achievement tests In addition to the SAT In applying 
for admission; but those achievement test are available only for that 
purpose. Some professional schools require all applicants to take a 
test » for example, the medical school and law school tests. Some 
graduate departments also require an achievement test In the 
discipline. All these test are constructed to serve a specific paying 
clientele. 

Standardized achievement tests in various fields reflect the 
collective judgments of professors from various colleges, and the 
writers of textbooks assigned to many students, about relevant content 
and understanding of the field. The use of such measures enables one to 
view student performance in a national perspective as welH as in the 
local perspective of the local college. 
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Most of these tests are measures of students' knowledge and 
understanding of an academic discipline. Since most courses are also 
organized by academic disciplines, the tests are relevant for the 
evaluation of Instruction; although obviously the specific objectives 
and content of a course taught by a specific professor to a specific 
class of students may differ In a good mar\y ways fran the content of the 
standardized test. Tests for evaluating broader general education goals 
are even less clearly related to Instruction. Moreover, there may be 
less agreement today than there was a few decades ago about what the 
content of such general achievement tests should be. Also, the 
published tests are norm-referenced to maximize finding reliable 
individual differences; whereas many current attitudes about testing 
favor canpetency -based, or content-based Instruments that seek to define 
mastery or minimum standards rather than sorting students along some 
distribution of scores. And further, the emphasis of some evaluators on 
using only measures specifically made for the objectives of a specific 
course is a barrier to the use of more general measures. 

For the evaluation of learning and instruction in a single course, 
the judgment of the professor is probably the best evidence we have, or 
can ever have, about students' mastery of the educational content. The 
best judge of what a student understands about economics is the 
professor of the economics course. The professor has observed the 
student in class, graded the mid-term and final exams, read the assigned 
papers. Moreover, the professor usually has years of experience, 
involving many students, in making these judgments. The professor's 
evaluation, expressed as a letter grade, is both reliable and valid 
within the reasonable limits of human frailty and dedication to 
fairness. By and large, then, there is nothing "wrong" with professors' 
ERIC • on 



grades as a measure of learning the content of a course. The criticisms 
of grades apply to the fact that they have an Individual and local 
definition, not a collective or national definition. An A at Swarthmore 
does not mean the same as an A at San Jose, because the grade reflects 
performance relative to the competition. This Is widely regarded as a 
criticism of grades as a criterion of learning; but. In fact. It should 
not be regarded as an Invalidation of the grade given by a professor to 
students In a particular course; It is, rather, an Invalidation of 
Interpreting grades gained under different conditions as If they were 
gained under the same conditions. 

For assessment, accountability, or accreditation purposes, student 
performance measures that are somewhat broader In scope than the 
objectives of a single Isolated course are needed. The referent Is the 
program or the college, and the quality that can be attributed to It 
rather than the Individual student In the single classroan. 

For this broader scope of evaluation a variety of new measures may 
be needed. Given the diversity of Institutions, curricula, and courses 
it Is unlikely that any one measure, or any required battery of 
measures, could cover the variety that exists. The more prudent, and I 
believe more realistic, approach is to encourage the construction of 
mar^ new measures and to suggest their use by any college that finds 
them interesting and relevant. For the remainder of this chapter I 
offer some suggestions about new measures. 
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1. A vocabulary test. Introductory courses Introduce students to 
the vocabulary of the field — a vocabulary which stands for Its basic 
concepts and elements. A vocabulary test could be viewed as measuring 
students' familiarity with the language of Intellectual discourse. One 
might think of a broadly educated person as one who has at least some 
understanding of what experts and scholars are saying! This would 
reflect a somewhat different view of general education. In an age of 
specialization and of the departmentalization of subject-matter, perhaps 
the generally educated person could be viewed as one who Is familiar 
with more than one specialization. In any case, a vocabulary test 
measuring students familiarity with the language of different academic 
disciplines should be relatively easy to construct. For example. In the 
UCLA conceptual diagrams of what an academic discipline Is all about, 
one can see examples of the language of the field, a language one needs 
to know If one Is to understand the field. 

Note the following words In the conceptual diagram for physics: 
electricity, magnetlom, light, particles, atoms, gases, solids, liquids, 
plasma, polymers, nucleus. Ionization, conductors. In the conceptual 
diagram for sociology, not Included here, I noted the following: social 
structuring, alienation, social norms, social status, social class, so- 
cial mobility, family, birth rate. In a conceptual diagram for psycho- 
logy I found such terms as perception, maturation, conditioning, uncon- 
scious, prejudice, cognition, emotions. In the conceptual diagram for 
biology, I noted these terms: ecology, habitat, neurons, cells, genes, 
proteins, chlorophyll. One could^ of course, readily Identify the 
fundamental terms of a field by examining the Introductory textbooks. 
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Suppose one constructed a vocabulary test that consisted of 10 
terms In each of five basic sciences {physics, chemistry, biology, 
astronomy, and geology), plus 10 terms In each of five basic social 
sciences (anthropology, sociology, economics, political science, and 
psychology), plus 10 terms In each of five areas In the huma^'ltles 
(language, literature, history, philosophy, and fine arts). This 150 
Item vocabulary test could be subdivided In various smaller weighted 
combinations, split Into equivalent forms, or arranged In other ways to 
fit the Interests of the college. It Is, In short, a flexible and 
adaptive measure. And presumably, whatever variations there are In the 
way courses are taught and In who teaches them, a course In physics 
would Involve a common basic vocabulary. 

To Illustrate the possible content of a vocabulary test I have 
jotted down two words under each of 15 disciplines — words which I 
suspect one might encounter In conversations or writings about the topic 
and which presumably one should know If one is to comprehend what the 
experts are talking about. 
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Science 

Chemistry 
carbon 
molecules 

Physics 
plasma 
nucleus 

Biology 
genes 
protein 

Astronomy 
galaxy 
light year 

Geology 
fossil 
crust 



Social Science 

Anthropol ogy 
cul ture 
mores 

Sociology 
class 

alienation 

Economics 
GNP 

tariff 

Political Science 
bureaucracy 
proletariat 

Psychol ogy 
emotions 
percepti on 



Humanities and Arts 

Language 
linguistics 
grammar 

LI terature 
satire 
plot 

Fine Arts 
abstract 
romantic 

HI story 

civilizations 
traditions 

Phil osophy 
loglT 

metaphysics 
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2. Understanding basic concepts. Whereas a vocabulary test might 
be constructed In a multiple-choice objective format, a test of 
understanding basic concepts could be In essay form; although It also 
could be In a multiple-choice format. By basic concept I mean beliefs, 
propositions, theories, etc. that have had a significant bearing on how 
man views the world. While these concepts have their roots In academic 
disciplines they have become very broadly Influential In many walks of 
life. Here are seme examples: the concept of relativity In physics, 
the concept of evolution In biology, the concept of ecology In viewing 
the environment, the concept of culture In anthropology, psychoanalytic 
concepts In psychology and psychiatry. How well can students describe 
and explain these concepts? 

3. Understanding major values. Liberal education Is typically 
described as concerned with fundamental values. Especially from 
literature and the humanities one presumably gains Insight about such 
values as truth, beauty, liberty, justice, freedan, loyalty, love, 
equality, etc. In a previous chapter I suggested that the admissions 
requirement of an English ccmposltlon could serve multiple purposes. 

The topics assigned fvor these compositions could Include ones that would 
reveal students understanding of Important values. One could ask '^What 
does freedom mean and how Is It exemplified In Individuals and society?" 
The same or similar topic could be assigned to seniors. One would hope 
that the meaning, range of application, and maturity of understanding 
exhibited In the responses of seniors would be better than In the 
responses of freshmen. The freshmen essays could be read first as 
English compositions, as they are now; but later read by a different 
group of judges (philosophers, historians, etc.) for the breadth and 
depth of understanding of the Idea or concept. 
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4. A current affairs test. This was done In the 1930's at 
Minnesota's General College, and subsequently published as a special 
feature In Time magazine. This doesn't specifically relate to college 
curricula, but It does exemplify a belief that knowledge of current 
events Is Important. The General College had a course titled 
Contemporary Affairs In which the basic readings were the dally 
Minneapolis or St. Paul newspaper, the Sunday New York Times, and Time 
magazine. One hundred of the Items written originally for the course 
exam were later printed In Time magazine, together with a scoring key 
(no peeking!), and a few general conments about the distributions of 
scores at the college — I.e. If you answered 85 or more of the 
questions correctly your score Is very, very good. The content of the 
current affairs test was divided Into specific topics national 
affairs. International affairs, science^, literature, arts, etc. If the 
idea of an annual current affairs test were revived, and developed In a 
W2iy that could get responses fran high school students, college 
students, and adults. It could becane a national Indicator of public 
awareness — similar to a Gallup poll but dealing with knowledge rather 
than opinions. 

5. A test that seeks to Integrate knowledge, attitude, and action. 
Most objectives of general liberal education are Integrative: for 
example, "to participate as an Informed and responsible citizen In 
accord with democratic Ideals*" We do not have any measures that deal 
with this sort of Integration within the Individual. A design for doing 
this was described In n\y book on Measuring Outcomes of College. This 
was presented as an Idea which someone might wish to develop. Last 
year, In connection with a research project, I put together a few Items 
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which, again, Illustrated the Idea. The sponsors of the research were 
Interested In exploring connections between knowledge about a topic, 
attitudes about 1t, and the sources of this knowledge and attitudes. 
The sponsoring agency was particularly Interested In the relationship 
between religious beliefs and attitudes about social Issues; so because 
of the religious Interest of the sponsor, test items were written in 
relation to Issues about which some underlying religious belief might be 
applicable — the environment, nuclear weapons, poverty, the family, 
materialism, etc. I have Illustrated one brief set of items here as an 
example of a test structure or design. The opportunity to develop and 
actually pretest the idea was not forthcoming because the project was 
canceled for financial reasons. 

In this example which follows there are three sets of questions: 
1) what do students believe is true about the topic? 2) what 
attitudes/values do they have? and 3) to what extent do their responses 
reflect experiences in school and/ or serious personal thought? 
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TOPIC: THE ENVIROJWENT 



BELIEFS 

What Is your belief about each of the following statements? 
one of the responses to the left of each statement. 



Check (x) 




1. In many places In the U.S. the underground water table 
Is being rapidly depleted. 

2. Pesticides, toxic wastes, etc. are polluting wells and 
other sources of drinking water. 

3. Air pollution fran Industry, and the fallout of acid 
rain, are destroying thousands of trees In the forests 
and fish In the lakes. 

4. So far, no safe way to dispose of radioactive materials 
has been found. 

5. The Increased use of plastics cremates Increased 
disposal problems because plastics are not biodegradable* 



ATTITUDES/VALUES 
Indicate your 
Check (x) one 



own attitudes/ values about each 
of the responses to the left of 



of the following 
each statement. 



statements. 




1. We win all survive or perish depending on how we use 
and conserve the resources of the Earth. 

2. An environment that will not support animals, fish, birds. 
Insects, and plants will not support people either. 

3. The Earth Is so big and bountiful that It can support Its 
human population for many generations Into the future. 

4. The Earth Is God's creation and we are responsible 
to God for the stewardship of Its resources. 

5. Science and Industry should not be allowed to produce 
materials that seriously pollute the land, sea, or air, 
even If that means unemployment for many people vAio now 
work In those Industries. 
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REFLECTIONS 

To what extent have any of your experiences In school (classes, readings, 
etc.) led you to think about and develop your views on this topic of 
THE ENVIROWENT? Check (x) one of the responses below. 

very much 

qui te a b1 1 
some 

very little 

Regardless of where your views may have been acquired, to \4iat extent do 
they reflect knowledge and serious thought on your part? Check (x) one of 
the responses below. 

very much a n 

"quite a bit 
some 

very 11ttle/I haven't really given much thought to this topic 



One could add another set of responses about personal actions related to 
the topic, such as: took old papers to a reycling center; belonged to a 
conservation group such the Sierra Club or the Audubon Society; etc. 

A test design similar to the one Illustrated for "The Environment" 
could provide professors and researchers with a new level of diagnostic 
Insight about students' understanding of educational content. 

The present discussion of current and new content measures has been 
limited to achievement tests. There are many other outcomes one should 
measure In evaluating the quality of students overall educational 
experiences — measures of attitudes, interests, values, and similar 
topics thr,t go beyond a set of courses but iny comments have concerned 
measures of content. Content defines what students are supposed to 
learn. The presumption Is that they will knoM more at the end of a 
course than they knew at the beginning, or perform a task better after 
Instruction and practice. This Is always true. It Is Impossible for a 
student to know more about King Lear before he has read the play than he 
will know after he has read It. One can. In fact, project this obvious 
truth to the entire course-taking experience In college. A college 
graduate will have satisfactorily completed about 30 to 45 different 
courses (depending on whether the college term Is divided Into two 
semesters or three quarters). In every one of those course^ the 
students' knowledge and understanding of the subject will have been 
demonstrated to the satlsfactlor^ of an expert judge (probably fneasured 
by course examinations); and In every case the student will know more 
about the subject at the end of the course than he knew at the 
beginning. A formal program of pre- and post- testing Is not necessary 
to prove the obvious. 
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To the question, "Who knows what about what outccmes?" surely one 
set of valid answers would come fran the people who are closest to the 
action. Those closest to the action are the actors themselves -- the 
professors and the students. In a monograph prepared for the Center for 
the Study of Evaluation at UCLA In 1985 (Pace and others, The 
Credibility of Student-Self Reports ) evidence was assembled showing 
that, with respect to many outcomes, students are very good judges of 
progress. The College Student Experiences Questionnaire (CSEQ) contains 
21 statements of goals and asks students to rate their progress or 
gain. Eight of those goals have some clear connection with college 
courses science, art, humanities. We know fran decades of 
achievement testing that majors know more about their major field than 
other students know. The students' ratings of progress are totally 
congruent with what we know from achievement test scores. So, when 
considering measures of achievement, we need to remind ourselves that 
professors and students are valid judges and reporters. 
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EXPANDING the RANGE of 
MEASURES of EDUCATIONAL PROCESSES and CONTEXTS 

Courses, content, programs, and curricula all exist In a context that 
Is larger than their own specific boundaries. The development, progress, 
and attainment of students is influenced by this larger context and by the 
processes with which learning and Instruction are carried out. The size 
and scope of the college or university Itself Is an example of this larger 
context. In some of the larger colleges and universities the liberal arts 
are a minority culture In a predominantly vocational environment. In some 
colleges and universities nearly all, or a large percent of students, live 
In college housing. In other places most students are conmuters. These 
are obvious Institutional features. The focus In this chapter on context 
and process Is on measures of the educational or psychological context, 
not on the physical or demographic features. The chapter begins by 
presenting several measures of this overall psychological context. Then, 
measures more explicitly fvcused on the process of teaching and learning 
are Illustrated. 

Some of the measures described In this chapter were originally part 
of a questionnaire that was answered by about 7300 upperclassmen In 79 
colleges In 1969. The results of this larger Inquiry have been 
published. (Pace. The Deir se of Diversity? A Comparative Profile of 
Eight Types of Institutions . Carnegie Commission, 1974.) Others are 
special Item combinations selected from College and University Environment 
Scales (CUES) Initially published by ETS In 1963. These measures and 
others were Included In the Higher Education Measurement and Evaluation 
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Kit, a loose-leaf collection of measures distributed to over 1000 colleges 
and unlvesltles In the years 1971-1975 by UCLA's Center for the Study of 
Evaluation. The Kit Is now out-of-print. We have selected, and In some 
cases slightly modified, some of that material for Inclusion In this 
report. The relevance of the measures to the assessment of quality In the 
Instructional and educational experience of undergraduates Is emphasized 
In the notes describing them. 

The College Environment 

College environments differ from one another. The character or 
atmosphere of a particular campus results from mar\y things the 
nature of Its student body and staff, the variety and emphasis of Its 
academic program. Its various pressures and expectations, prevailing 
customs, procedures, and policies, and Its relationship to political 
and cultural events In the larger society. An Instrument for 
measuring these differences, the College and University Environment 
Scales (CUES) was used by several hundred schools following Its 
Initial publication by the Educational Testing Service In 1963. The 
complete Instrument Is no longer available, but selected Items could 
still be used to characterize the college environment. 

Students who are familiar with the environment from having lived 
In It for more than a year serve as reporters. Indicating If, In their 
experience and perception, the condition or event described by each of 
the statements Is "true" about their college. When there Is consensus 
among the reporters by a margin of two to one or greater, the 
statement Is regarded as being "characteristic" of the campus. 
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The statements selected for this brief environment scale can be 
^ grouped under four dimensions: 

1. Practicality - an environment characterized by enterprise, 
organization, material benefits, and social activities; 
vocational and collegiate emphases; orderly supervision, 
(items 1-4) 

2. Community - a friendly, cohesive, group-oriented campus, 
(items 5-8) 

3. Awareness - campus encourages concern about social and 
political problems, expressiveness through the arts, 
tolerance of criticism, (items 9-12) 

4. Scholarship - an environment characterized by intellectuality 
and scholastic discipline. Intellectual achievement, and the 
pursuit of knowledge, (items 13-16) 

In each item, the response TRUE is consistent with the dimension 
it defines. This abbreviated version of CUES is illustrated here. The 
items selected are reasonably reflective of the range of content in each 
of the longer scales of the original instrument. 
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THE COLLEGE ENVIR0M4ENT 

Directions: Facilities, procedures, policies, requirements, attitudes, 
etc., differ fran one campus to another. What is characteristic of your 
campus? As you read each of the statements below, check the space under 
TRUE (T), if the statement describes a condition, event, attitude, etc. 
that is generally characteristic of your college; or under FALSE (F) if it 
is not generally characteristic of the college. Please answer every 
statement. 

Generally 
T F 

^_ 1. Frequent tests are given in most courses. 

2. The college offers many really practical courses such as 
typing, report writing, etc. 

— — _ 3. The most important people at the school expect others to show 
proper respect for them. 

4. There is a recognized group of student leaders on campus. 

5. Mar\y upperclassmen play an active role in helping new students 
adjust to campus life. 

6. The professors go out of their way to help you. 

7. The school has a reputation for being friendly. 

_^_^8. It's easy to get a group together for card games, singing, 
going to the movies, etc. 

9. Students are encouraged to criticize administrative policies 
and teaching practices. 

10. The school offers many opportunities for students to under- 
stand and criticize important works in art, music, and drama. 

1 1. Students are actively concerned about national and 
international affairs. 

12. Many famous people are brought to the campus for lectures, 
concerts, student discussions. 

1 3. Most courses are a real intellectual challenge. 

1 4. Students set high standards of achievement for themselves. 

15. Most courses require intensive study and preparation out of 
class. 

^16. Careful reasoning and clear logic are valued most highly in 

grading student papers, reports, or discussions. 
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Campus Morale 

The Campus Morale scale presents another dimension for describing 
the context or setting In which learning and development are promoted. 
Items canprlsing this measure have also been drawn from College & 
University Environment Scales. 

Following an analysis of the kinds of Items that have been used 
In the study of morale In other organizations— mainly military and 
Industrial— we developed an outline of the most common content. Next 
we examined the 150 Items In CUES and Identified 55 whose content ap- 
peared to be relevant to the concept of morale. We then made statis- 
tical and psychometric studies of hew well these Items held together 
In what might be regarded as a common scale. The final result Is a 
set of 22 Items that can be appropriately used to produce an Indica- 
tion of Campus Morale. This morale scale Is not a measure of Indivi- 
dual student morale; It Is a measure of campus morale. Operationally 
the judgment about Campus Morale Is the number of "morale-relevant" 
statements which are seen as characteristic of the Institution, defin- 
ed as consensus among reporters by a margin of two to one or greater. 

Further analyses of the 22 Items have shown that they can be 
grouped Into five categories or factors, as follows: 1) students' 
freedom of expression, 2) assimilation Into campus life, 3) group 
coheslveness, 4) commitment to Intellectual goals, and 5) Identifi- 
cation with social norms. A high morale campus describes an environ- 
ment characterized by acceptance of social norms, group coheslveness, 
friendly assimilation Into campus life, and also a commitment to 
Intellectual pursuit and freedom of expression. Intellectual goals 
are exemplified and widely shared In an atmosphere of personal and 
social relationships that are both supportive and spirited. 
For each Item the high morale response Is Indicated by 
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CmP\}$ MORALE 

Directions: Facilities, procedures, policies, requirements ^ attitudes, 
etc., differ fran one campus to another. What Is characteristic of your 
campus? As you read each of the statements beloM, check the space under 
TRUE (T), If the statement describes a condition, event, attitude, etc., 
that Is generally characteristic of your college; or under FALSE (F) If It 
Is not generally characteristic of the college. Please answer every 
statement. 

Generally 

T F 

J( 1. The big college events draw a lot of student enthusiasm and 

support. 

L 2. Anyone who knows the right people in the faculty or 

administration can get a better break here. 

J( 3. The professors go out of their way to help you. 

J( 4. Students have many opportunities to develop skill in 

organizing and directing the work of others. 

J( 5. Many upperclassmen play an active role in helping new students 

adjust to campus life. 

J( 6. When students run a project or put on a show everybodly knows 

about it. 

J(^^^ 7. Students exert considerable pressure on one another to live 
up to the expected codes of conduct. 

J( 8. There is a lot of group spirit. 

X 9. Most of the faculty are not interested in students' personal 
problems. 

X 10. The school helps everyone get acquainted. 

11. Channels for expressing students' ccmplaints are readily 
accessible. 

j( 12. A controversial speaker always stirs up a lot of student 

discussion. 
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CAMPUS MORALE (continued) 



X 13. Many students here develop a strong sense of responsibility 
about their role In contemporary social and political life. 

X 14. The expression of strong personal belief or conviction Is 
pretty rare around here. 

X 15. There Is considerable Interest In the analysis of value 
systems, and the relativity of societies and ethics. 

X 16. Students are conscientious about taking good care of school 

property. 

— JL Students pay little attention to rules and regulations. 

X 18. Many students seem to expect other people to adapt to them 
rather than trying to adapt themselves to others. 

X 19. Most of the professors are very thorough teachers and really 
probe Into the fundamentals of their subjects. 

X 20. Students set high standards of achievement for themselves. 

X 21. Students put a lot of energy Into everything they do— In 

class and out. 

X 22. Most courses are a real Intellectual challenge. 
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Environmental Emphases 

The College Student Experiences Questionnaire (CSEQ) Includes a 
section labeled The College Environment. There are eight rating scales 
for students to Indicate their perceptions if the environment. Five of 
those ratings are about the environmental press or emphasis which 
students believe the college gives to certain lines of student 
development. This portion of the CSEQ Is reproduced below: 
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THE COLLEGE ENVIROfMENT 



Colleges differ froii one another In the extent to v^lch they emphasize or 
stress various aspects of students' development. Thinking of your own 
experience at this college, to what extent do you feel that each of the 
following Is emphasized? The responses are numbered from 7 to 1, with the 
highest and lowest points described. Fill In the space of whichever number 
best Indicates your Impression on this seven-point rating scale. 



Emphasis on the development of academic, 
scholarly, and Intellectual qualities 



Strong emphasis 



Emphasis on the development of esthetic, 
expressive, and creative qualities 



Strong enphasls 



Emphasis on being critical, 
evaluative, and analytical 



Strong emphasis 



Emphasis on the development of vocational 
and occupational competence 



Strong emphasis 



Emphasis on the personal relevance 
and practical values of your courses 



Strong emphasis 



Weak emphasis 



Weak emphasis 



Weak emphasis 



Weak emphasis 



1 Weak emphasis 
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These characterlz^itlons of environmental emphases are further examples of the 
context for student learning and development. 
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style of Learning: Academic 

The general inanner In which students pursue their academic work 
differs from one campus to another. There are also obvious 
differences among students on any one campus, related to their 
personal characteristics and to the subjects they are studying. 

The Items In this scale measure the degree of "academic 
involvement and Intensity" — the style of one's effort In relation to 
the acquisition of knowledge and understanding fran courses and 
readings, such as participation In class discussions, talking with 
professors, devoting concentrated periods of time to academic work, 
and reading related but unasslgned materials. A high score on this 
scale signifies a high level of scholarly Intensity and participation 
In academic life. 

Although the scale In Its present form has not been pre-tested, 
the content was suggested by previous study In which a panel of 20 to 
25 students at each of four quite different colleges and universities 
kept a detailed timelog of academic activities for a period of one 
week. From that study it was found that certain types of activities 
were much more common on some campuses than on others, and that these 
differences seemed to reflect a particular learning style and 
Intensity. 

Some of the Items from this scale have subsequently been used In 
Pace's College Student Experiences Questionnaire. 
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STYLE OF LEARNING: ACADEMIC 


Directions: Please check (X) each activity you engaged In during 
the most recent "typical" week. 


During a typical week: 


1. 


I participated In a class discussion. 




1 nao a conversation, lasxing a nair-nour ur lunyer} wim 
one or more of n\y professors. 


3. 


I discussed with other students for an hour or longer the 
subject-matter of one or more of n\y courses. 


4. 


I spent a concentrated period of time— three hours or 
longer wixnoux interruption" -5 tuajing ror one ui hut 
courses. 


5. 


I studied at least four hours or longer during the 
weekend. 


6. 


I read a book related to one of wy courses but thai was 
not an assigned reading for the course. 


7. 


I spent five or more hours writing papers. 


8. 


I spent some time just browsing In the library or 
bookstore. 


9. 


I participated In a research project. 


^10. 


I spent five hours or more looking up references In the 
library and taking notes. 
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style of Learning: Experiential 

The college years provide students with many opportunities for 
personal and social development- -meeting new people, encountering new 
social situations, and facing problems of personal choice. Some 
people believe that one way to make education more "relevant" Is to 
give greater recognition to the value and significance of direct 
personal experiences, making self -understanding, social. Insight, and 
personal growth relevant educational objectives. Student Interest In 
sensitivity training, encounter groups, urban problems, minorities, 
ghetto children, student-faculty relations, etc. are pertinent to this 
aspect to education. Some of these activities may not be as common 
today as they were In the 1960s. 

The Items In the scale have not been pre-tested. The scale was 
developed to provide a counterbalance to the scale on academic 
learning, and to enable colleges to gain systematic knowledge about 
the nature and extent of these other kinds of learning experiences. 
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STYLE OF LEARNING: EXPERIENTIAL 



Directions: Check (X) each activity you have engaged In during the 
past term. 

During the past term: 

1. I participated in a sensitivity group. 

2. I became well acquainted, personally and socially, with a 

student whose race Is different than mine. 

3. I became well acquainted with a foreign student. 

^ 4. I did seme work with children or parents In a neighborhood 

different from mine. 

5. I had one or more long and private conversations with a 

professor or counselor. 

6. I was In an Informal social gathering where students and 
faculty members came to know one another personally 
(not just superficially). 

7. I had the experience of being In a marathon encounter 

group. 

8. I helped a student who was having difficulty In a course. 
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Educational Experiences & Preferences 

Students prior educational experiences and preferences for 
different styles of learning have a bearing on what and how they are 
likely to learn best. With many new modes of teaching, and with the 
diversity of students In higher education, the effectiveness of 
teaching and learning within a college Is partly a problem of 
providing enough variability In the Instructional forms It offers to 
accommodate students of the various styles (abilities, Intellectual 
disposition, personality traits) It enrolls. 

One personal characteristic, variously described as 
authoritarian vs. non-authoritarian, rigidity vs. flexibility, 
dependence vs. Independence Is related to Intellectual disposition and 
to preferences for certain kinds of learning experiences. The brief 
questionnaire we have composed Is not described as a personality test 
and has not been validated as such. Nevertheless, It Is Indirectly a 
personality preference for highly explicit vs. more open-ended 
educational activity, with the Items equally divided between 
experiences that are highly structured and explicit and experiences 
that are more flexible, autonomous, or open-ended. 

The questionnaire could be used as a pre-test for estimating the 
selective attraction of students to an experimental program compared 
with a traditional program. It could also be used as a post-test to 
see whether any change In preferences Is associated with new 
experiences. And It could be used In determining the 
Interrelationships among experience, preference, and performance In a 
broader Inquiry regarding the effectiveness of Instruction. 
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EDUCATIONAL EXPERIENCES & PREFERENCES 

During the past year in school (high school or college) about how often 
did you have each of the experiences listed below, and how well did you 
like (or would you like) that type of course work or a:tivity? Please 
check (X) in the appropriate columns. 



How often have 
you had this 
experience? 



How wel 1 do you 
like (or would 
you like) this 
type or course 
work or activity? 




1. Having very clear and specific 
assignments. 

2. Giving an oral report to a class. .^^^ 

3. Having frequent tests so as to know 
exactly how well I was achieving. 

4. Taking an independent study course 

where I had to plan my own work. , 

5. Being told exactly v^en assignments 

or projects must be completed. ^_ 

6. Being called on in class to express 

ny opinion about a topic. 

7. Following a definite schedule of 

time for study. 

8. Working on a group project with 

other members of the class. 

9. Taking exams in which the answers 
can be clearly graded as right or 

wrong. 

10. Having to present arguments on 

both sides of an issue. 



0) 
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EDUCATIONAL EXPERIENCES & PREFERENCES (continued) 



How often have 
you had this 
experience? 



How wel ] do you 
like (or would 
you like) this 
type or course 
work or activity? 



11. Memorizing specific facts and 
terms for an exam. 



CO 



12. Planning and carrying out a 
research study In which I had 
to collect and Interpret data. 

13. Having an Instructor who clearly 
Indicates In his lectures what 
the main points are so that It 
Is easy to take notes. 

14. Participating In a class 
discussion In which various 
students express their opinions. 

15. Writing a report or term paper 
on a topic assigned by the 
Instructor. 



16. Writing a report or term paper 
on a topic of n\y own choosing. 
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Report of Course Activities and Attitudes 

One of the common objections to maoy student rating scales of 
courses and Instructors Is that the format and the questions appear to 
put the student in the role of judge, a role that assumes an appro- 
priate and discerning basis for making the judgments. It is possible 
to evaluate courses from a different perspective -- one that regards 
students as reporters about their activities, Interests, and feelings 
rather than as judges about course content, methods, and values. 

Presumably, a course and the teacher of It hope to engage a 
student's Interest, activity, and Intelligence. To v<hat degree and 
with how many students have these engagements occurred? Intellectual 
engagement can range from simple to complex — from merely memorizing 
facts and terms one Is told to understand to higher level skills 
Involving application, analysis, and synthesis. Interest and 
attltudlnal Involvement can range from disinterest and rejection, to 
receptivity, to desiring, and to enjoyment and satisfaction. The 
level of activity and enthusiasm can range from passive and private, 
to open participation, and to public advocacy. 

Parallel sets of Items, reflecting a range from minimal to 
maximal levels of Involvement, have been developed for each of these 
three typies of responses — Intellectual (Items 1-9), affective 
(Items 10 - 18), and behavioral (Items 19 - 25). Each Item set Is 
Intended to form a scale which can be scored to show the degree of 
student Involvement that has presumably been stimulated by the course 
and Its Instructor. Moreover, since the Items are quite specific, the 
answer can be correspondingly explicit and accurate; and obviously the 
student himself Is the only qualified reporter about his own behavior. 
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The conceptual base and example for these scales canes from the 
Taxonomies of Educational Objectives, by Benjamin Bloom, David 
Krathwohl, and others. Just as cognitive objectives range fron simple 
to complex, so also the level and quality of students' learning 
activities range frcm simple memory of details to more Integrative and 
analytical learning. This Inherent parallelism formed the 
psychological rationale In the taxonomy of cognitive objectives; and a 
corresponding type of parallelism Is Implicit In the taxonany of 
affective objectives. Ti^e items In the present questionnaire 
described as behavioral (Items 19 - 25) could be regarded as belonging 
equally veil to the so-called affective scale (Items 10 - 18). 
However, It may be preferable and perhaps more diagnostic to 
conceptualize three dimensions of the quality of learning experience: 
simple to complex; disinterest to Interest; and passive to active. 

While one could use these scales Instead of the more common 
rating scales of courses and Instructors, it would be better and more 
Informative to use them as a supplement to other measures because they 
appear to provide a different perspective rather than simply beini a 
substitute technique. 

It was the successful try-out of this scale, and the positive 
response of the faculty members who used it, that led to the 
construction of the quality of effort scales in Pace's CSEQ. If the 
concept works with respect to course activities, v^y couldn't it be 
developed with respect to other aspects of campus experience? 
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REPORT OF COURSE ACTIVITIES AND ATTITUDES 

During this course about how often did you do each of the following 
things? (Check the appropriate space before each of the activities listed 
below.) 



1. Took detailed notes in class or on reading 
assignments. 

2. Memorized facts, vocabulary, and terminology. 

3. Underlined major points In the readings. 

4. Made outlines from class notes or readings. 

5. Attempted to explain the material to another 
student or friend. 

6. Thought about applications of the material to 
other situations. 

7. Tried to relate the material to Ideas and 
experiences of my own. 

8. Looked for some basic structure or organization 
In the material . 

9. Tried to see how different facts and Ideas fit 
together. 

10. Postponed doing work related to the course. 

11. Skipped class. 

12. Listened attentively In class meetings. 

13. Participated In class discussions. 
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REPORT OF COURSE ACTIVITIES AND ATTITUDES (continued) 




14. Voluntarily did work that was not required. 

15. Enjoyed working in this course. 

16. Thought I would like more courses in this field. 

17. Thought I would like courses in fields related to 
this. 

18. Asked the instructor for additional materials to 
study and think about. 

19. Talked with other students in the course about 
the topics being studied. 

20. Talked with the instructor after class. 

21. Made appointments with the instructor to talk 
about the course. 

22. Had meetings with other students who were 
particularly interested in the course. 

23. Told other students (not in the class) about the 
interesting materials or ideas in the course. 

24. Told friends outside the college about the 
interesting materials or ideas in the course. 

25. Recommended the course to other students. 
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Quality of Teaching & Faculty -Student Relations 



When thinking about the attributes of a good college professor, 

the following might come to mind: 

The professor Is 

a dedicated scholar, 
a thorough teacher, 

and sets high standards of achievement. 

In his courses he 

keeps his materials up-to-date, 
clearly explains the goals and purposes, 
and stimulates good discussions. 

In his relations with students, he Is 
helpful , 
friendly, 

and Interested In them as Individuals. 
It Is these attributes that are measured In this brief scale on this 
topic. The Items are drawn from College and University Environment 
Scales (Educational Testing Service, 1963, 1969) and are brought 
together here as a separate test. On a campus In which good teaching 
and supportive faculty-student re'^atlonshlps are characteristic, itiost 
students would agree that many of the above attributes are generally 
true of their professors and courses. The "desirable" response to 
each Item Is Indicated by X . 
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THE PROFESSORS 



Directions: As you read each of the statements below, think of 
yourself as a reporter. What Is generally characteristic about the 
professors and courses at your college? Check the space under TRUE 
(T) If the statement describes a condition, event, or activity that 
Is generally characteristic of your college; or under FALSE (F) If 
it IS not generally characteristic of the college. Please answer 
every statement. 

Generally 

T F 



X 



X 



1. Most of the professors are dedicated scholars in their 
— field. 

2. Courses, examinations and readings are frequently 

revised. 

X 3. Personality, pull, and bluff get students through many 
courses. 

4. The professors go out of their way to help you. 

5. 



7. 

8. 

X 10. 
11. 



Most of the professors are very thorough teachers and 
really probe into the fundamentals of their subjects. 

Faculty members rarely or never call students by their 
first names. 

Instructors clearly explain the goals and purposes of 
their courses. 

Most of the faculty are not Interested in students' 
personal problems. 

Standards set by the professors are not particularly 
hard to achieve. 

Students almost always wait to be called on before 
speaking in class. 



j( 11. Class discussions are typically vigorous and intense. 
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Supportive Relationships 

The college environment ratings In the CSEQ Include three that ask 
students to characterize the Interpersonal relationships among students 
at a college, between students and faculty members, and with admlnstrative 
personnel. These measures are reproduced below: 



THE COLLEGE ENVIROWENT 

The next three ratings refer to relationships among people at the college. 
Again, thinking of your own experience, how would you rate these 
relationships on the seven-point scales? 



Relationship with other students, 
student groups, and activities 



riendly. Supportive, 
Sense of belonging 



Competitive, Un1nvolved| 
Sense of alienation 



Relationships with faculty members 
7 6 5 4 3 2 1 



Approachable, Helpful, 
Understanding, Encouraging 



Remote, Discouraging, | 
Unsympathetic 



Relationships with administrative 
personnel and offices 



1 



Helpful, Considerate, 
Flexible 



Rigid, Impersonal, 
Bound by regulations! 
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General Satisfaction with College 

If people don't like their present circumstances, would rather be 
somewhere else, and feel that what they are doing Is unimportant, it 
would be quite safe to conclude that they are probably wasting their 
time and not taking advantage of opportunities available to them* 
Enthusiasm, satisfaction, and a sense of worthwhileness are personal 
feelings that contribute to productivity and accomplishment. Most 
students In most colleges like being there and believe that what they 
are doing Is Important and beneficial. The results from various 
surveys of college students support this generally favorable 
judgment. Nevertheless, the proportion of dissatisfied students Is 
greater on some campuses than on others. The brief "satisfaction 
Index" provides a reliable estimate of this aspect of the educational 
context. This Index was used In Pace's 1969 survey of upperclassmen. 
The first two Items are also Included In the CSEQ. 
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GENERAL SATISFACTION 
WITH COLLEGE 



1. How well do you like college? (Check One) 

^l. I don't like It. 

2. I am more or less neutral about It. 

3 . I like It. 

4. I am enthusiastic about It. 



2. If you could start over again, would you go to the same 
college you are now attending? (Check One) 

^l. No, definitely 

2. Probably no 

3. Probably yes 

4. Yes, definitely 



3. Regardless of any vocational benefit college may have for 
you, do you think that being In college at this time In 
your life is a very Important and beneficial experience? 
(Check One) 

^l. Definitely no 

2 . Generally no 
3. Generally yes 

4. Definitely yes 
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The varljus measures described In this chapter can be classified 
along two dimensions. Some are measures of process; and some are 
measures of the context In which the processes occur. Second, some 
are measures oriented to the Individual Instructor and course; and 
some are oriented to education or Instruction In general. For 
example, context measures Include The College Environment, Campus 
Morale, and the ratings of Env v ,'onmental Emphases. Process measures 
are Illustrated by Styles of Learning, Educational Experiences and 
Preferences, and the Report of Course Activities and Attitudes, the 
Professors, Supportive Relationships, and Satisfaction. The Report of 
Course Activities and Educational Experiences and Preferences are 
directly applicable to and useable by an individual Instructor. The 
other measures refer mainly to education and instruction in general. 

Overall, the measures Illustrated here are relevant to several 
widely acknowledged good practices in education: For example, the 
Importance of active learning, of adaptations to individual 
differences, of good faculty-student contacts, and of holding high 
expectations for student performance. 
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BEYOND THE COLLEGE YEARS: ENRICHING THE CONTENT OF ALUMNI STUDIES 



The emphasis In this report on expanding the range of measures of 
content and achievement, on better measures of the context in which 
learning and development occurs, and on the processes of learning and 
Instruction all reflect tn underlying concern with the problems of 
estimating the outcomes of euucatlon and Instruction. The focus has 
been on measuring outcomes during the college years. In this chapter 
we look beyond the college years for additional ways of judging 
outcomes. 

Most alumni surveys have focused on recent graduates. The 
content has emphasized occupational and financial status, job 
satisfaction, the relation of jobs to major fields of study In 
college, and subsequent success In and preparation for graduate study 
or other further education. ACT has an alumni survey questionnaire 
that deals with these topics. But these topics - jobs, money, and 
graduate work - are surely not reflective of some of the main purposes 
of higher education. Higher education, more fundamentally. Is 
concerned with "critical thinking, the clarification of one's 
philosophy, athlcs, and morality, with responsible citizenship, 
esthetic sensitivity, tolerance, appreciation of other cultures, 
self-directed learning, understanding science and technology, 
vocational training, and breadth of knowledge" (Pace, Measuring 
Outcomes of College . 1979, p. 110). In this book Pace goes on to 
suggest content for alumni studies that has not usually been Included: 
evidence of knowledge possessed, personal achievement, general life 
satisfaction. Intellectual interests and habits, standards and 
Involvement In civic and cultural affairs, broad perspectives about 
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critical Issues, problems, and choices, reflections on their own 
education, and views about the Importance of higher education as a 
social Institution. Questions at this level of significance are 
appropriate for alumni 10 to 20 years after graduation. 

The farther removed one Is from the college experience, the more 
difficult It Is to "prove" that one's current status was caused by the 
Influence of college. But this presumed difficulty Is partly owing to 
the fact seme researchers will not believe anything unless the 
research Is done their way. The data have to be longitudinal; there 
must be a control group; one must discount the results In relation to 
students' basic abilities, family backgrounds, and prior 
achievements. In short, nothing can be accepted as true unless It has 
been demonstrated by a particular methodolr y. This, of course. Is 
nonsense; and It results In judgments about education that are 
fundamentally false. The fact Is, from all comprehensive alumni 
surveys, that the pattern of activities. Interests, attitudes, etc. of 
adults 20 years after graduation is remarkably similar to the patterns 
of interest, curricular emphasis, and knowledge that characterized 
their college experience. Put somewhat differently one can say that 
the areas of knowledge, interest, and values that were emphasized in 
college are still evident in adult behavior 20 years later. This is 
not coincidence; it reflects a continuity in life history and is also 
consistent with the college experience. Its cause cannot be 
attributed to SAT scores, or high school grades, or to becoming 20 
years older. "Proof" in the experimental definition is not 
applicable. One does not need to undertake alumni studies for the 
purpose of trying to prove the benefit of higher education, for many 
of the benefits are self-evident and universally recognized, 
especially those related to occupation. 
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This chapter in the present report Is not a list of all the 
things one might Include In an alumni questionnaire. Rather, Its main 
content and purpose Is to suggest some new categories of questions and 
modified ways of asking questions that have been asked in prior 
studies. Its purpose is to add to the Ideas one might consider In 
developing an alumni questionnaire intended for alumni 10 to 20 years 
after graduation. Some of these Ideas were originally developed by 
the writer during a term as a Visiting Scholar at Virginia Military 
Institute In 1980. Further developments of the Ideas were presented 
to the Association of America Colleges in 1984, and to Georgetown 
University in 1986 for their consideration and possible use. 
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Abilities & Skills - College Emphasis and Personal Gain 



The abilities and skills described In the following set of 
questions are ones commonly included in lists of educational 
objectives and often found in survey questionnaires. One could add or 
subtract a few in case that seemed desirable. The abilities fall into 
five categories, with two or more specific examples in each. Sane of 
these skills and abilities may have 'Sittle connection with the formal 
curriculum or with specific subject-matter, but ar - nevertheless 
abilities valued by higher education and by socle in general • items 
12, 13, 14, 15, for example. Previous surveys of students as well as 
alumni show that there are differences in the responses to these items 
related to the 1;ype of college and the major field. Also, I think it 
is true that certain institutions would give much more attention or 
emphasis than others would to seme of these abilities. 

The double set of ratings - college emphasis and personal gain - 
would enable one to examine attainments in relation to objectives. 
This would be an Improvement over most past questionnaires that 
typically ask only about gains or benefits. 

This set of ratings could also be given to seniors on the verge 
of graduation. Comparisons between their responses and what the 
graduates recall about them 10 or 20 years later might be instructive. 
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COLLEGE EMPHASIS AND PERSONAL GAIN 




Listed below are various types of abilities or skills. Sane of them, 
although not necessarily all, are often described as among the goals or 
purposes of higher education - goals such as critical tninKing, effective 
canmuni cation, etc. As you now look back on your own undergraduate college 
experience, about how much emphasis (opportunity and encouragement) was 
there at your college on developing these abilities? And, about how much 
do think you personally Improved or gained In these various abilities? 
Indicate your opinions by circling the appropriate number defined as 
follows: 


College emphasis 
5 > very much emphasis 
4 * quite a bit of empnasis 
3 - some/moderate emphasis 
2 - little empnasis 
1 - no emphasis 
0 - no opinion, don't know 


Personal Improvement or gain 
5 » very much gain 
4 » quite a bit of gain 
3 » some/moderate gain 
2 « little gain 
1 s no gain 

0 no opinion, don't know 


Abilities and Skills 


College Emphasis 


Personal Gain 


Ability to: 
1. Put Ideas together, see 

relationships, similarities & 
differences between Ideas 


5 4 3 2 10 


5 4 3 2 10 


2. Think analytically & logically 


5 4 2 1 0 


5 4 3 2 10 


3. Apply knowledge to new problems 


5 4 3 2 10 


5 4 3 2 10 


4. Find Information you need 


5 4 3 2 10 


5 4 3 2 10 


AD1 1 1 ty lo: 
5. Write well 


5 4 3 2 1 0 


5 4 3 2 1 0 


6. Speak well 


5 4 3 2 10 


5 4 3 2 10 


7. Use quantitative tools 
(statistics, etc.) 


5 4 3 2 1 0 


5 4 3 2 1 0 


8. Communicate through artistic and 
creative expressions 


5 4 3 2 1 0 


5 4 3 2 1 0 


Ability to: 
9. Recognize and cope with moral and 
ethical Issues 


5 4 3 2 1 0 


5 4 3 2 10 


10. Put current problems In historical, 
cultural, philosophical perspective 


5 4 3 2 1 0 


5 4 3 2 1 0 


11. Evaluate and choose between 
alternative courses of action 


5 4 3 2 1 0 


5 4 3 2 10 
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COLLEGE EMPHASIS AND PERSONAL GAIN (continued) 



Abilities and Skills 



Ability to: 

12. Function as a team member 

13. Organize A supervise tasks and 
groups of people 

14. Sense the feelings 4 perception 
of other people 

15. Relate well to people of different 
races, nations, & religions 

Ability to: 

16. Understand yourself - your 
abilities & limitations. Interests 
& personality 

17. Acquire new skills and knowledje 
on your own 



College Empahsis 

5 4 3 2 10 
5 4 3 2 10 

5 4 3 2 1 0 

5 4 3 2 1 0 

5 4 3 2 1 0 

5 4 3 2 1 0 



Personal Gain I 

5 4 3 2 10 
5 4 3 2 1 0 

5 4 3 2 1 0 

5 4 3 2 1 0 

5 4 3 2 1 0 

5 4 3 2 1 0 
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Areas of Knowledge - Influences and Importance 

This next set of items assumes that one of the intentions of 
higher education Is (or ought to be) to make people knowledgeable 
about what 1s going on in the world now - i.e., to develop students' 
awareness of new developments, new thinking, etc.. In different 
categories of knowledge such as science and technology, the arts and 
humanities, and the social sciences. 

Answers to the first set of ratings (Influences of college) 
will probably reflect primarily whatever the student's major field 
was* However, "current events" Is not a course In the college 
curriculum; and It Is quite possible that at some colleges little 
influence Is exerted on students to keep up on such matters, and 
perhaps little stimulus to develop Interest. Whatever emphasis 
colleges and universities choose to give (and I think must give) to 
knowledge about what has been, or more broadly the cultural heritage, 
there must also be (In iny opinion) a parallel and complementary 
concern with what's happening In the world we now live In. I suspect 
some Institutions exemplify this concern much better and more 
effectively than other. Public support for higher education Is 
probably related strongly to public belief in Its Importance to life 

and society today. 

From the second set of ratings (Importance of knowledge for work 
and for personal satisfaction), one can explore many Interesting 
relationships; fo*' example: 1) whether the type of knowledge 
important for work Is the same as the type of knowledge acquired in 
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college; 2) If not, then what are the main lines of difference?; 3) 
whether knowledge and Interests developed In college are also 
Important for personal satisfaction later, regardless of their utility 
in employment; 4) who Is more interested In current events and social 
problems - those whose education and work is mainly science oriented, 
or those who are mainly arts and humanities oriented? These are just 
sane of the connections that can be traced. 

One obvious Intent here is simply to see what areas of knowledge. 
If ar\y, give people the greatest personal satisfaction. There will 
be, I hope, some evidence that college graduates are concerned about 
what's going on in the world, and that they find it personally 
important as well as satisfying to be aware of new developments and 
new thinking, whatever their line of employment may be, or whatever 
their major field of study In college may have been. 
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ARFAS OF KNOWLEDGE 

At the left hand side of this page there Is a list of six different areas or types of knowledge. In each of these 
areas of knowledge indicate first how much Influence your undergraduate college experience had In your acquiring such 
knowledge and Interest. Then, on the right hand side of the page. Indicate how Important this type of knowledge Is 
for the kind of work you now do, and for your personal Interest and satisfaction. Make your response by circling the 
number that expresses your rating according to the following definitions: r ^ ^ 



Influence of college on acquisition/Interest 
5 « very much Influence 
4 ' quite a bit of Influence 
3 » seme Influence 
2 = little Influence 
1 " no Influence 



Importance 
5 
4 
3 
2 

1 = 



AREAS OF KNOWLEDGE 



1. New developments In science 
- new concepts, theories 



Influence of your college 
experience In acquiring 
knowledge and developing 
Interest In this area 



of knowledge for woric /personal satisfaction 
absolutely essential 
very Important 
generally Important 
marginally Important 
not Important 



Importance of this type of knowledge 
For the kind of For your personal 
work you do new Interests and satlsfacti 



2. New developments & Inventions in 
technology - electronics, energy, 
transportation, medical technology, etc. 

3. Knowledge of v^at contemporary drama- 
tists, poets, & artists are producing 

4. Major current events In the news 
at home and around the world 

5. Different political positions related 
to social problems such as the economy, 
the law, welfare, defense, etc. 

6. Knowledge of what leading thinkers, 
d historians, etc. are writing about 

EKLC people, values, cultures, and ways of life 

I 



5 
5 

5 
5 



4 
4 

4 
4 



3 
3 

3 
3 



2 
2 

2 
? 



1 
1 

1 

1 



4 3 



5 4 3 2 1 



on 



5 4 3 2 1 

5 4 3 2 1 

5 4 3 2 1 

5 4 3 2 1 

5 4 3 2 1 

5 4 3 2 1 



5 4 3 2 1 

5 4 3 2 1 

5 4 3 2 1 

5 4 3 2 1 

5 4 3 2 1 



5 4 3 2 1 
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Range of Knowledge 

Varieties of knowledge is something that has not been tapped in 
previous studies, at least so far as I am aware. It would be nice, I 
suppose, to transfonn all this into a genuine multiple-choice 
achievement test. But then I'm not sure people would answer it. So, 
rv« settled on the notion of asking them whether they think they 
could define/describe/explain the concepts/terms/ instruments. One 
might wonder about the credibility of their answers; but past research 
suggests that one need not worry. The types of knowledge are 
obviously related to broad and specific subject-matter studied or not 
studied in college. What they say they can do will be closely related 
to what we know they have studied. The interesting analysis to be 
made is an exploration of breadth (who has it?). 

Also, of course, one should take a careful look at the content of 
these four sets and add or subtract as desired. One should decide 
what ideas or concepts are especially pertinent to a college's 
objectives, and perhaps modify the lists accordingly. What is 
illustrated here is a type of item that may be very useful. It is an 
alumni level counterpart to one of the earlier suggestions for 
expanding the range of outcome measures for undergraduates. 
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RANGE OF KNOWLEDGE 

In every field of knowledge there are, frcm time to time, new explanations, 
concepts, discoveries that change the way we view the universe, the earth, 
our Institutions, and ourselves* Listed belcw are some words denoting such 
concepts and phenomena In the sciences and the social sciences. If someone 
said to you, "What Is this?" What could you say? For each Item Indicate 
your response by using the following definitions. 

I could define/ describe It clearly, specifically, and correctly. 

I could define/ describe It In general terms, generally correct. 
I could give a vague explanation, but not very adequate. 
I really couldn't answer the question. 

i 

Gravity 
Relativity 
Radiation 
Energy 

Photosynthesis 
Evolution 
DNA 

Tectonic plates 



Marxism . 

Psychoanlysis 

Conditioning 

Corporation 

Bureaucracy 

The market 

Gross National Product 
Cultural relativity 



In literature and the arts there are words that Indicate different forms or 
styles of expression. Listed below are some of those words. If someone 
said to you, "What Is this?" What could you say? For each Item Indicate 
your response by using the following definitions: 



I could define It clearly, with lots of Illustrations. 

I could define It In general terms, with some examples. 

I could Indicate the general Idea, but couldn't Illustrate It 
exactly. 

I really couldn't answer the question. 

i 

Satire 
Irony 
Epic 
Myth 

Metaphor 
Anal ogy 

Abstract cf? 
Representative 



RANGE OF KNOWLEDGE (continued) 



Many of the things we take for granted in our technically sophisticated 
world, because they are so familiar, are things many of us don't really 
understand. How do they work? What makes them do what they do? How, for 
example, does a picture get onto your TV set? Listed below are some 
scientific/technical Instruments that In various ways affect our lives. If 
someone said to you, "How does this work?" What could you say? For each 
Item Indicate your response by using the following definitions: 



V 



could explain It accurately - both theory and application. 
I could give an explanation In general terms. 

I could indicate some of what Is Involved but not technically. 

II really couldn't answer the question, 
i 

Electric light 
Automobile engine 
Telephone 
Phonograph 
Radio 

TV picture 
X-raiy 

Electronic canputer 
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Civic and Cultural Affairs 



Fran the Alumni Survey Questionnaire, 1969, I've selected a few 
Items from each of the activity scales. In each case the Items 
represent different levelo of civic and cultural Involvement. If 
these or other Items frcm the 1969 survey were used, sane Interestlrvg 
comparisons could be made. I've also suggested a few questions about 
reading books and magazines. 

The Intent of these Items is to assess the level and range of 
Involvement In activities that have some relevance to the objectives 
of higher education. 
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CIVIC AND CULTURAL ACTIVITIES 



READING 

Not counting what you read for 
your work, about how mar\y books 
of fiction did you read during 
the past year? 
none 

one or two 
t hree or four 
between five and nine 
ten or more 



Not counting what you read for 
your work, about how many books 
of non-fiction did you read 
during the past year? 
n one 

^one or two 

three or four 
between five and nine 
ten or more 



Not counting professional or 
technical magazines 
specifically related to your 
work, about how many magazines 
do you subscribe to and/or 
read regularly? 
n one 

o ne or two 

three or four 

five or more 



In the following categories of 
activities, ched ach of the 
activities you .e engaged In 
during the past year . 



COMMUNITY AFFAIRS 

During the past year: 

^I followed local events 

regularly In my newspaper. 
I gave money to the community 
fund or chest or other local 
charity. 

^I belonged to a canmunlty organ- 
ization Interested In civic 
affairs - such as PTA, Chamber 
of Commerce, League of Women 
Voters, business or 
professional association, etc. 



NATIONAL and STATE POLITICS 

During the past year: 

I listened to speeches, news 
specials, discussion programs, 
Qtc» about political Issues on 
TV or radio weekly or monthly 

^I followed state and national 

political events regularly 
In iny newspaper. 

^ I voted In the last state 

election. 

^I contributed money to some 

political club or group. 
I signed a petition, wrote as 

~" letter, card, or telegram 

concerned with some political 
Issue. 
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CIVIC AND CULTURAL ACTIVITIES (continued) 



ART 

During the past year: 

I talked about art with iny 

friends. 
I attended an exhibition of 

contemporary painting or 

sculpture. 
I bought a palntinci or piece 

of sculpture. 



LITERATURE 

During the past year: 

^I talked about new books with 

my friends. 
^I bought books for my personal 

library. 
I read poetry. 



RELIGION 

During the past year: 

I belonged to a church. 

I contributed a regular sum of 

money to the church. 
I discussed Ideas, practices, or 

problems of religion with my 

friends. 



INTERCULTURAL 
During the past year: 

I talked with my friends about 
people and cultural events In 
other countries. 

I saw one or more foreign movies. 

I read one or more books by 
authors from another country. 



EDUCATION 

During the past year: 

^ I read about education In the 
newspaper. 
I voted (or would vote) In favor 
of a bond Issue or other pro- 
position to provide more money 
for the public schools. 

^I enrolled In a course offered 

by a college or university. 



MUSIC 

During the past year: 

I talked about music with my 
friends. 

^I attended one or more symphony, 

opera, or chamber music 
concerts. 
I attended one or more concerts 
of contemporary folk music, 
rock, Jazz, etc. 



SCIENCE 

During the past year: 

^I talked about science with n(y 

friends. 

^I read articles about new deve- 
lopments in scientific research 
in the newswpapers or magazines 

_^_I attended a scientific exhibit or 
"" museum. 



INTERNATIONAL 

During the past year: 

discussed international rela- 
tions, foreign policy, the U.N., 
etc. with my friends. 

^I read newspapers or magazine 

articles dealing with 
international relations. 
I wrote to a news publication or 
government official in behalf of 
some legislation or U.S. policy 
regarding international 
relations. 
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DRMA 

During the past year: 

^I talked about movies, plays, TV 

dramas, etc. with my friends. 
I attended one or more plays -- 
either professional or amateur, 

saw several movies that could be des- 
cribed as experimental, avant garde, etc. 



These suggestions for the content of alumni studies Illustrate the 
fact that at this level of testing one Is no longtr assessing student 
learning and development In college. One is, however, exploring current 
Interest, activities, and knowledge that are adult counterparts to 
certain broad aspects of the college experiences; and to that extent one 
Is Identifying residual and enduring Impressions and Influence. 
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SUMMARY AND SUGGESTIONS 



The emphasis on assessment, accreditation, and accountability 
makes the measurement of outcomes and judgments about quality In 
higher education topics of widespread discussion and concern. This 
concern Is evident at various levels — federal, state, regional. 
Institutional, and Individual. 

Sane people believe there should be a national Index of 
educational quality, or, more r^eallstlcally, several Indicators 
which would be a profile of quality. Perhaps this would be 
analogous to econanic Indicators, such as GNP, Dow-Jones, Consumer 
Price Index, etc. But there are many relevant Indicators of the 
econanic quality of the nation — unemployment, percent of families 
living In poverty, national debt, trade balances — and they do not 
add up to a consistent conclusion. Dow-Jones goes up, but so does 
the national debt. Despite the Ingenuity of econcmlsts In devising 
Indicators there remains considerable dispute about what some of 
the Indicators really mean. There would, I suspect, be even more 
dispute about the Interpretation of educational Indicators; and 
still more dispute about their appropriateness. 

At the stute level, the concern for accountability Is often 
expressed by setting minimum standards of performance, especially 
In basic skills such as reading, writing, math, etc. Sane states 
have aU. :"ov1ded financial Incentives to encourage local 
evaluations. 

Often one finds the suggestion that "gains" In students 
knowledge, understanding, skills, etc. should be emphasized in 
judging the quality of Instruction and the college experience. 
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The term "value added" is often used to express this, and sane 
writers have proposed that the quality of an Institution's educa- 
tion for undergraduates should be judged by how much students gain 
between entrance and exit. The only thing new about "value added" 
Is the terra Itself. Education has always been concerned with stu- 
dent learning; and the proof of learning has always been evidence 
of change, or a demonstrated capacity to perform at some desired 
level. The problem of "measuring" change, however. Is no simple 
matter. This Is not the place for a discussion of the technical 
Issues In the measurement of change. Three problems can be cited 
briefly: 1) the error of measurement Is magnified when one 
compares the differences between two scores each of which has Its 
own error of measurement; 2) the potential for change Is related to 
one's position In the distribution of scores on the Initial test; 
3) the same test Items, encountered a second time, may not be 
measuring the same thing they Initially measured. So, our existing 
testing technology Is of doubtful adequacy for measuring change; 
and, the conclusions one draws fran such technology are of dubious 
validity. Many changes are obvious and do not require fonnallzed 
testing. And for many objectives, students themselves may be the 
only valid source for estimating "gains." 

The regional accrediting associations recognize the Importance 
of Institutional differences, and require self-studles designed to 
clarify purposes, examine programs, and measure productivity. This 
Individualized approach, together with general guidelines, allcws 
the college to examine Its own strengths and weaknesses. Today, 
what is popularly called assessment also typically recognizes the 
Importance of local purposes and programs. 
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To some extent, assessment Is an encouragement for more 
colleges to do more publicly and more systematically what some 
colleges have done periodically for maiv years -- through the 
college's Institutional research office, the records of Its 
students on such exams as GRE, various questionnaire surveys, 
locally made examinations, and other college records. At the 
beginning of this present report, I expressed the view that the 
assessment of Institutional quality must consider all the alms and 
activities of the Institution, and that no single Indicator of 
quality was equally appropriate for all institutions. This Is also 
true, I believe. In the assessment of educational and Instructional 
quality. Consequently, iny own efforts to facilitate the evaluation 
of educational and Instructional activities have usually been 
toward expanding the range and variety of measures that might be 
used rather than to devise or recommend a single measure to apply 
to everyone. 

An adequate assessment of educational and Instructional 
quality requires a battery of measures, a variety of observations, 
and the recognition of multiple alms. The measures described In 
this report are a step In this direction. What needs to be given 
much greater attention Is the role that students themselves can 
have In evaluation — perhaps by keeping time logs, diaries, 
writing an autobiography, and other activities that directly 
express and document not only the process of learning but also the 
product. Self-evaluation, under guidance, would contribute to 
personal learning and development In the same way that 
Institutional self-study contributes to Institutional understanding 
and development. 
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